Methods and systems for correcting image misalignment

ABSTRACT

The invention provides methods of determining a correction for a misalignment between at least two images in a sequence of images due at least in part to sample movement. The methods are applied, for example, in the processing and analysis of a sequence of images of biological tissue in a diagnostic procedure. The invention also provides methods of validating the correction for a misalignment between at least two images in a sequence of images of a sample. The methods may be applied in deciding whether a correction for misalignment accurately accounts for sample motion.

PRIOR APPLICATIONS

[0001] The present application is a continuation-in-part of U.S. patentapplication Ser. No. 10/068,133, filed Feb. 5, 2002, which is acontinuation of U.S. patent application Ser. No. 09/738,614, filed Dec.15, 2000, which claims priority to and the benefit of U.S. ProvisionalPatent Application Serial No. 60/170,972, filed Dec. 15, 1999; thepresent application also claims the benefit of the co-owned U.S.Provisional Patent Application entitled, “Methods and Systems forCorrecting Image Misalignment”, by Schott et al., filed on Sep. 30,2002, and referenced by attorney docket number MDS-031PR. All of theabove applications are assigned to the common assignee of thisapplication and are hereby incorporated by reference.

FIELD OF THE INVENTION

[0002] This invention relates generally to image processing. Moreparticularly, the invention relates to correcting image misalignment,where the misalignment is due at least in part to sample movement.

BACKGROUND OF THE INVENTION

[0003] In modern medical practice, it is useful to analyze a sequence ofimages of in vivo tissue obtained throughout the course of a diagnosticmedical procedure. For example, in screening for some forms of cervicalcancer, a chemical agent is applied to cervical tissue and the opticalresponse of the tissue is captured in a sequence of colposcopic images.The tissue is characterized by analyzing the time-dependent response ofthe tissue, as recorded in the sequence of images. During this type ofdiagnostic procedure, the tissue may move while images are being taken,resulting in a spatial shift of the tissue within the image frame field.The tissue movement may be caused by the natural movement of the patientduring the procedure, which can occur even though the patient attemptsto remain completely still. Accurate analysis of the sequence of imagesmay require that the images be adjusted prior to analysis to compensatefor misalignment caused at least in part by patient movement.

[0004] There is currently a method of stabilizing an electronic image bygenerating a motion vector which represents the amount and direction ofmotion occurring between consecutive frames of a video signal. See U.S.Pat. No. 5,289,274 to Kondo. However, this method accounts for certaingross movements of a video camera—in particular, certain vibrationscaused by the operator of a handheld camcorder. The method does notcompensate for misalignment caused by movement of a sample. For example,such a method could not be used to adequately correct an imagemisalignment caused by the small-scale movement of a patient during adiagnostic procedure.

[0005] Another image stabilization method is based on detecting thephysical movement of the camera itself. See U.S. Pat. No. 5,253,071 toMacKay, which describes the use of a gimbaled ring assembly that movesas a camera is physically jittered. These types of methods cannot beused to correct misalignments caused by the movement of a sample.

SUMMARY OF THE INVENTION

[0006] The invention provides methods of correcting misalignmentsbetween sequential images of a sample. The invention is particularlyuseful for correcting image misalignment due to movement of the samplebetween images and/or during image acquisition. The invention alsoallows for real-time, dynamic image alignment for improved opticaldiagnosis and assessment.

[0007] In a preferred embodiment, the invention comprises determining anx-displacement and a y-displacement corresponding to a misalignmentbetween two images of a tissue sample, where the misalignment is causedby a shift in the position of the sample with respect to the image framefield. For example, in obtaining a sequence of images of an in-situtissue sample, an embodiment of the invention makes it possible tocorrect for small image misalignments caused by unavoidable patientmotion, such as motion due to breathing. It has been discovered thatvalidating misalignment corrections improves the accuracy of diagnosticprocedures that use data from sequential images, particularly where themisalignments are small and the need for accuracy is great. Thus,methods of the invention comprise validating misalignment corrections bysplitting individual images into smaller subimages, determiningdisplacement between these subimages, and comparing the subimagedisplacements to the overall image displacement. Alternatively,validation may comprise adjusting two images according to a misalignmentcorrection, then determining displacement between correspondingsubimages and comparing these displacements with a threshold maximumvalue.

[0008] It has also been discovered that application of a chemicalcontrast agent, such as acetic acid, prior to or during acquisition of asequence of tissue images enhances the detection of small-scale imagemisalignment by increasing intra-image contrast of the tissue images.The enhanced contrast of the tissue features recorded in the imagesallows for more accurate motion correction determination, since enhancedfeatures may serve as landmarks in determining values of displacement.

[0009] Both misalignment correction determination and validation may beperformed such that an accurate adjustment is made for a misalignmentbefore an entire sequence of images is obtained. This allows, forexample, “on the fly” adjustment of a camera while a diagnostic exam isin progress. Thus, corrections may be determined, validated, andaccurately adjusted for as misalignments occur, reducing the need forretakes and providing immediate feedback as to whether an examination iserroneous. Automatic adjustment may be accomplished by adjusting aspectsof the optical interrogation of the sample using a misalignmentcorrection value. Adjustments may be performed, for example, byadjusting aspects of transmission and/or reception of electromagneticenergy associated with the sample. This may include, for example,transmitting a correction signal to a galvanometer system or a voicecoil to “null out” a misalignment by adjusting the position of a mirroror other component of the camera obtaining the images according to thecorrection signal. Alternatively, or additionally, adjustments may beperformed by electronically adjusting an aspect of an image, forexample, the frame and/or bounds of an image, according to amisalignment correction value, or by performing any other appropriateadjustment procedure.

[0010] Applications of methods of the invention include the processingand analysis of a sequence of images of biological tissue. For example,chemical agents are often applied to tissue prior to optical measurementin order to elucidate physiological properties of the tissue. In oneembodiment, acetic acid is applied to cervical tissue in order to whitenthe tissue in a way that allows enhanced optical discrimination betweennormal tissue and certain kinds of diseased tissue. The acetowhiteningtechnique, as well as other diagnostic techniques, and the analysis ofimages and spectral data obtained during acetowhitening tests aredescribed in co-owned U.S. patent application Ser. No. 10/099,881, filedMar. 15, 2002, and co-owned U.S. patent application entitled, “Methodand Apparatus for Identifying Spectral Artifacts,” identified byAttorney Docket Number MDS-033, filed Sep. 13, 2002, both of which arehereby incorporated by reference.

[0011] A typical misalignment between two images is less than about0.55-mm within a two-dimensional, 480×500 pixel image frame fieldcovering an area of approximately 25-mm×25-mm. These dimensions providean example of the relative scale of misalignment versus image size. Insome instances it is only necessary to compensate for misalignments ofless than about one millimeter within the exemplary image frame fielddefined above. In other cases, it is necessary to compensate formisalignments of less than about 0.3-mm within the exemplary image framefield above. Also, the dimensions represented by the image frame field,the number of pixels of the image frame field, and/or the pixelresolution may differ from the values shown above.

[0012] A misalignment correction determination may be inaccurate, forexample, due to any one or a combination of the following:non-translational sample motion such as rotational motion, localdeformation, and/or warping; changing features of a sample such aswhitening of tissue; and image recording problems such as focusadjustment, missing images, blurred or distorted images, lowsignal-to-noise ratio, and computational artifacts. Validationprocedures of the invention identify such inaccuracies. The methods ofvalidation may be conducted “on-the-fly” in concert with the methods ofdetermining misalignment corrections in order to improve accuracy and toreduce the time required to conduct a given test.

[0013] Once an image misalignment is detected, an embodiment providesfor automatically adjusting an optical signal detection device, such asa camera. For example, a camera may be adjusted “on-the-fly” tocompensate for misalignments as images are obtained. This improvesaccuracy and reduces the time required to conduct a given test.

[0014] The optical signal detection device comprises a camera, aspectrometer, or any other device which detects optical signals. Theoptical signal may be emitted by the sample, diffusely reflected by thesample, transmitted through the sample, or otherwise conveyed from thesample. The optical signal comprises light of wavelength falling in arange between about 190-nm and about 1100-nm. One embodiment comprisesobtaining one or more of the following from one or more regions of thetissue sample: fluorescence spectral data, reflectance spectral data,and video images.

[0015] Methods comprise analysis of a sample of human tissue, such ascervical tissue. Methods of the invention also include analysis of othertypes of tissue, such as non-cervical tissue and/or nonhuman tissue. Forexample, methods comprise analysis of one or more of the following typesof tissue: colorectal, gastroesophageal, urinary bladder, lung, skin,and any other tissue type comprising epithelial cells.

[0016] A common source of misalignment is movement of a sample. Methodscomprise the steps of: obtaining a plurality of sequential images of asample using an optical signal detection device; determining acorrection for a misalignment between two or more of the sequentialimages, where the misalignment is due at least in part to a movement ofthe sample; and compensating for the misalignment by automaticallyadjusting the optical signal detection device.

[0017] The two or more sequential images may be consecutive, or they maybe nonconsecutive. In one embodiment, a misalignment correction isidentified between a first image and a second image, where the secondimage is subsequent to the first image. The first image and second imagemay be either consecutive or nonconsecutive.

[0018] Identifying a misalignment correction may involve data filtering.For example, some methods comprise filtering a subset of data from afirst image of a plurality of sequential images. A variety of datafiltering techniques may be used. In one embodiment, Laplacian ofGaussian filtering is performed. Identifying a misalignment may comprisepreprocessing a subset of data from the first image prior to filtering.For example, color intensities may be converted to gray scale beforefiltering. In some embodiments, filtering comprises frequency domainfiltering and/or discrete convolution in the space domain.

[0019] In order to identify a correction for a misalignment, preferredembodiments comprise computing a cross correlation using data from eachof two of the plurality of sequential images. In some embodiments,computing a cross correlation comprises computing a product representedby F_(i)(u,v) F*_(j)(u,v), where F_(i)(u,v) is a Fourier transform ofdata derived from a subset of data from a first image, i, of theplurality of sequential images, F*_(j)(u,v) is a complex conjugate of aFourier transform of data derived from a subset of data from a secondimage, j, of the plurality of sequential images, and u and v arefrequency domain variables. In preferred embodiments, the computing ofthe cross correlation additionally comprises computing an inverseFourier transform of the product represented by F_(i)(u,v)F*_(j)(u,v).

[0020] A method of the invention comprises validating a correction for amisalignment determined between a first image and a second image.Validating a misalignment correction comprises defining one or morevalidation cells within a bounded image plane; computing for eachvalidation cell a measure of displacement between two (or more) imagesbound by the image plane using data from the two images corresponding toeach validation cell; and validating a correction for misalignmentbetween the two images by comparing the validation cell displacementswith the correction. Preferably, each validation cell comprises a subsetof the bounded image plane. The two (or more) images may be consecutiveimages. In some embodiments, the validating step includes eliminatingfrom consideration one or more measures of displacement forcorresponding validation cells. For example, measures of displacementfrom validation cells determined to be likely to contribute to anerroneous validation result are eliminated in some embodiments. In someembodiments, identifying validation cells that are likely to contributeto an erroneous validation result comprises calculating a sum squaredgradient for at least one validation cell.

[0021] Methods of the invention comprise obtaining a plurality ofsequential images of the sample during an application of a chemicalagent to the sample. For example, the chemical agent comprises at leastone of the following: acetic acid, formic acid, propionic acid, butyricacid, Lugol's iodine, Shiller's iodine, methylene blue, toluidine blue,indigo carmine, indocyanine green, and fluorescein. Some embodimentscomprise obtaining sequential images of the sample during anacetowhitening test.

[0022] In preferred embodiments, the movement of the sample is relativeto the optical signal detection device and comprises at least one of thefollowing: translational motion, rotational motion, warping, and localdeformation.

[0023] One or more of the sequential images comprise measurements of anoptical signal from the sample. The optical signal comprises, forexample, visible light, fluoresced light, and/or another form ofelectromagnetic radiation.

[0024] Methods of the invention comprise determining a correction formisalignment between each of a plurality of pairs of images. Suchmethods comprise the steps of: obtaining a set of sequential images of asample using an optical signal detection device; and determining acorrection for a misalignment between each of a plurality of pairs ofthe sequential images, where at least one of the misalignments is due atleast in part to a movement of the sample. The correction may then beused to compensate for each of the misalignments by automaticallyadjusting the optical signal detection device.

[0025] The obtaining step and the determining step may be performedalternately or concurrently, for example. One embodiment comprisesdetermining a correction for a misalignment between a pair of thesequential images less than about 2 seconds after obtaining the latterof the pair of the sequential images. In another embodiment, this takesless than about one second.

[0026] In another aspect, the invention is directed to a method ofdetermining a correction for a misalignment that includes validating thecorrection. Methods comprise the steps of: obtaining a plurality ofsequential images of a sample using an optical signal detection device;determining a correction for a misalignment between at least two of thesequential images; and validating the correction for misalignmentbetween two of the images. An embodiment further comprises compensatingfor the misalignment by automatically adjusting the optical signaldetection device according to the correction determined. In oneembodiment, determining a misalignment correction between two images andvalidating the correction is performed in less than about one second.

[0027] Methods of the invention comprise compensating for a misalignmentby determining a correction for a misalignment between a pair of images,validating the misalignment, and automatically realigning one of thepair of images. The realignment may be performed during the acquisitionof the images, or afterwards.

BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The objects and features of the invention can be betterunderstood with reference to the drawings described below, and theclaims. The drawings are not necessarily to scale, emphasis insteadgenerally being placed upon illustrating the principles of theinvention. In the drawings, like numerals are used to indicate likeparts throughout the various views.

[0029]FIG. 1A represents a 480×500 pixel image from a sequence of imagesof in vivo human cervix tissue and shows a 256×256 pixel portion of theimage from which data is used in determining a correction for amisalignment between two images from a sequence of images of the tissueaccording to an illustrative embodiment of the invention.

[0030]FIG. 1B depicts the image represented in FIG. 1A and shows a128×128 pixel portion of the image, made up of 16 individual 32×32 pixelvalidation cells, from which data is used in performing a validation ofthe misalignment correction determination according to an illustrativeembodiment of the invention.

[0031]FIG. 2A is a schematic flow diagram depicting steps in a method ofdetermining a correction for a misalignment between two images due to atleast in part to the movement of a sample according to an illustrativeembodiment of the invention.

[0032]FIG. 2B is a schematic flow diagram depicting steps in a versionof the method shown in FIG. 2A of determining a correction for amisalignment between two images due to at least in part to the movementof a sample according to an illustrative embodiment of the invention.

[0033]FIG. 2C is a schematic flow diagram depicting steps in a versionof the method shown in FIG. 2A of determining a correction for amisalignment between two images due to at least in part to the movementof a sample according to an illustrative embodiment of the invention.

[0034]FIG. 3 depicts a subset of adjusted images from a sequence ofimages of a tissue with an overlay of gridlines showing the validationcells used in validating the determinations of misalignment correctionbetween the images according to an illustrative embodiment of theinvention.

[0035]FIG. 4A depicts a sample image after application of a 9-pixel size(9×9) Laplacian of Gaussian filter (LoG 9 filter) on an exemplary imagefrom a sequence of images of tissue according to an illustrativeembodiment of the invention.

[0036]FIG. 4B depicts the application of both a feathering technique anda Laplacian of Gaussian filter on the exemplary unfiltered image used inFIG. 4A to account for border processing effects according to anillustrative embodiment of the invention.

[0037]FIG. 5A depicts a sample image after application of a LoG 9 filteron an exemplary image from a sequence of images of tissue according toan illustrative embodiment of the invention.

[0038]FIG. 5B depicts the application of both a Hamming window techniqueand a LoG 9 filter on the exemplary unfiltered image used in FIG. 5A toaccount for border processing effects according to an illustrativeembodiment of the invention.

[0039]FIG. 6 depicts the determination of a correction for misalignmentbetween two images using methods including the application of LoGfilters of various sizes, as well as the application of a Hamming windowtechnique and a feathering technique according to illustrativeembodiments of the invention.

DESCRIPTION OF THE ILLUSTRATIVE EMBODIMENT

[0040] In general, the invention provides methods of determining acorrection for a misalignment between images in a sequence due tomovement of a sample. These methods are useful, for example, in thepreparation of a sequence of images for analysis, as in medicaldiagnostics.

[0041] In some diagnostic procedures, methods of the invention compriseapplying an agent to a tissue in order to change its optical propertiesin a way that is indicative of the physiological state of the tissue.The rate and manner in which the tissue changes are important in thecharacterization of the tissue.

[0042] Certain embodiments of the invention comprise automated andsemi-automated analysis of diagnostic procedures that have traditionallyrequired analysis by trained medical personnel. Diagnostic procedureswhich use automatic image-based tissue analysis provide results havingincreased sensitivity and/or specificity. See, e.g., co-owned U.S.patent application Ser. No. 10/099,881, filed Mar. 15, 2002, andco-owned U.S. patent application entitled, “Method and Apparatus forIdentifying Spectral Artifacts,” identified by Attorney Docket NumberMDS-033, filed Sep. 13, 2002, both of which are incorporated herein byreference.

[0043] In order to facilitate such automatic analysis, it is oftennecessary to adjust for misalignments caused by sample movement thatoccurs during the diagnostic procedure. For example, during a givenprocedure, in vivo tissue may spatially shift within the image framefield from one image to the next due to movement of the patient.Accurate diagnosis requires that this movement be taken into account inthe automated analysis of the tissue sample. In some exemplaryembodiments, spatial shift correction made at the time images areobtained is more accurate than correction made after all the images areobtained, since “on-the-fly” corrections compensate for smaller shiftsoccurring over shorter periods of time, rather than larger, morecumulative shifts occurring over longer periods of time.

[0044] If a sample moves while a sequence of images is obtained, theprocedure may have to be repeated. For example, this may be because theshift between consecutive images is too large to be accuratelycompensated for, or because a region of interest moves outside of ausable portion of the frame captured by the optical signal detectiondevice. It is often preferable to compensate for misalignments resultingfrom sample movement during the collection of images rather than waituntil the entire sequence of images has been obtained beforecompensating for misalignments. Stepwise adjustment of an optical signaldetection device throughout image capture reduces the cumulative effectof sample movement. If adjustment is made only after an entire sequenceis obtained, it may not be possible to accurately compensate for sometypes of sample movement. On-the-fly, stepwise compensation formisalignment reduces the need for retakes.

[0045] On-the-fly compensation may also obviate the need to obtain anentire sequence of images before making the decision to abort a failedprocedure, particularly when coupled with on-the-fly, stepwisevalidation of the misalignment correction determination. For example, ifthe validation procedure detects that a misalignment correctiondetermination is either too large for adequate compensation to be madeor is invalid, the procedure may be aborted before obtaining the entiresequence of images. It can be immediately determined whether or not theobtained data is useable. Retakes may be performed during the samepatient visit; no follow-up visit to repeat an erroneous test isrequired. A diagnostic test invalidated by excessive movement of thepatient may be aborted before obtaining the entire sequence of images.

[0046] In preferred embodiments, a determination of misalignmentcorrection is expressed as a translational displacement in twodimensions, x and y. Here, x and y represent Cartesian coordinatesindicating displacement on the image frame field plane. In otherembodiments, corrections for misalignment are expressed in terms ofnon-Cartesian coordinate systems, such as biradical, spherical, andcylindrical coordinate systems, among others. Alternatives toCartesian-coordinate systems may be useful, for example, where the imageframe field is non-planar.

[0047] Some types of sample motion—including rotational motion, warping,and local deformation—may result in an invalid misaligment correctiondetermination, since it may be impossible to express certain instancesof these types of sample motion in terms of a translationaldisplacement, for example, in the two Cartesian coordinates x and y. Itis noted, however, that in some embodiments, rotational motion, warping,local deformation, and/or other kinds of non-translational motion areacceptably accounted for by a correction expressed in terms of atranslational displacement. The changing features of the tissue, as inacetowhitening, may also affect the determination of a misalignmentcorrection. Image recording problems such as focus adjustment, missingimages, blurred or distorted images, low signal-to-noise ratio (i.e.caused by glare), and computational artifacts may affect the correctiondetermination as well. Therefore, validation of a determined correctionis often required. In some embodiments, a validation step includesdetermining whether an individual correction for misalignment iserroneous, as well as determining whether to abort or continue the testin progress. Generally, validation comprises splitting at least aportion of each of a pair of images into smaller, corresponding units(subimages), determining for each of these smaller units a measure ofthe displacement that occurs within the unit between the two images, andcomparing the unit displacements to the overall displacement between thetwo images.

[0048] In certain embodiments, the method of validation takes intoaccount the fact that features of a tissue sample may change during thecapture of a sequence of images. For example, the optical intensity ofcertain regions of tissue change during an acetowhitening test.Therefore, in preferred embodiments, validation of a misalignmentcorrection determination is performed using a pair of consecutiveimages. In this way, the difference between the corresponding validationcells of the two consecutive images is less affected by gradual tissuewhitening changes, as compared with images obtained further apart intime. In some embodiments, validation is performed using pairs ofnonconsecutive images taken within a relatively short period of time,compared with the time in which the overall sequence of images isobtained. In other embodiments, validation comprises the use of any twoimages in the sequence of images.

[0049] In some exemplary embodiments, a determination of misalignmentcorrection between two images may be inadequate if significant portionsof the images are featureless or have low signal-to-noise ratio (i.e.are affected by glare). Similarly, validation using cells containingsignificant portions which are featureless or which have lowsignal-to-noise ratio may result in the erroneous invalidation of validmisalignment correction determinations in cases where the featurelessportion of the overall image is small enough so that it does notadversely affect the misalignment correction determination. For example,analysis of featureless validation cells may produce meaninglesscorrelation coefficients. One embodiment comprises identifying one ormore featureless cells and eliminating them from consideration in thevalidation of a misalignment correction determination, therebypreventing rejection of a good misalignment correction.

[0050] A determination of misalignment correction may be erroneous dueto a computational artifact of data filtering at the image borders. Forexample, in one exemplary embodiment, an image with large intensitydifferences between the upper and lower borders and/or the left andright borders of the image frame field undergoes Laplacian of Gaussianfrequency domain filtering. Since Laplacian of Gaussian frequency domainfiltering corresponds to cyclic convolution in the space-time domain,these intensity differences (discontinuities) yield a large gradientvalue at the image border, and cause the overall misalignment correctiondetermination to be erroneous, since changes between the two images dueto spatial shift are dwarfed by the edge effects. Certain embodimentsemploy pre-multiplication of image data by a Hamming window to remove orreduce this “wraparound error.” Preferred embodiments employimage-blending techniques such as feathering, to smooth any borderdiscontinuity, while requiring only a minimal amount of additionalprocessing time.

[0051]FIG. 1A represents a 480×500 pixel image 102 from a sequence ofimages of in vivo human cervix tissue and shows a 256×256 pixel portion104 of the image from which data is used in identifying a misalignmentcorrection between two images from a sequence of images of the tissue,according to an illustrative embodiment of the invention. Preferredembodiments comprise illuminating the tissue using either or both awhite light source and a UV light source. The image 102 of FIG. 1A has apixel resolution of about 0.054-mm. The embodiments described hereinshow images with pixel resolutions of about 0.0547-mm to about0.0537-mm. Other embodiments have pixel resolutions outside this range.In some embodiments, the images of a sequence have an average pixelresolution of between about 0.044-mm and about 0.064-mm. In theembodiment of FIG. 1A, the central 256×256 pixels 104 of the image 102are chosen for use in motion tracking. Other embodiments use regions ofdifferent sizes for motion tracking, and these regions are notnecessarily located in the center of the image frame field. In theembodiment of FIG. 1A, the method of motion tracking determines anx-displacement and a y-displacement corresponding to the translationalshift (misalignment) between the 256×256 central portions 104 of twoimages in the sequence of images.

[0052] The determination of misalignment correction may be erroneous forany number of various reasons, including but not limited tonon-translational sample motion (i.e. rotational motion, localdeformation, and/or warping), changing features of a sample (i.e.whitening of tissue), and image recording problems such as focusadjustment, missing images, blurred or distorted images, lowsignal-to-noise ratio, and computational artifacts. Therefore, inpreferred embodiments, validation comprises splitting an image intosmaller units (called cells), determining displacements of these cells,and comparing the cell displacements to the overall displacement. FIG.1B depicts the image represented in FIG. 1A and shows a 128×128 pixelportion 154 of the image, made up of 16 individual 32×32 pixelvalidation cells 156, from which data is used in performing a validationof the misalignment correction, according to an illustrative embodimentof the invention.

[0053]FIG. 2A, FIG. 2B, and FIG. 2C depict steps in illustrativeembodiment methods of determining a misalignment correction between twoimages of a sequence, and methods of validating that determination.Steps 202 and 204 of FIG. 2A depict steps of developing data from aninitial image with which data from a subsequent image are compared inorder to determine a misalignment correction between the subsequentimage and the initial image. An initial image “o” is preprocessed 202,then filtered 204 to obtain a matrix of values, for example, opticalintensities, representing a portion of the initial image. In oneembodiment, preprocessing includes transforming the three RGB colorcomponents into a single intensity component. An exemplary intensitycomponent is CCIR 601, shown in Equation 1:

I=0.299R+0.587G+0.114B  (1)

[0054] where I is the CCIR 601 “gray scale” intensity component,expressed in terms of red (R), green (G), and blue (B) intensities. CCIR601 intensity may be used, for example, as a measure of the “whiteness”of a particular pixel in an image from an acetowhitening test. Differentexpressions for intensity may be used, and the choice may be geared tothe specific type of diagnostic test conducted. In an alternativeembodiment, a measure of radiant power as determined by a spectrometermay be used in place of the intensity component of Equation (1). Someembodiments comprise obtaining multiple types of optical signalssimultaneously or contemporaneously; for example, some embodimentscomprise obtaining a combination of two or more of the followingsignals: fluorescence spectra, reflectance (backscatter) spectra, and avideo signal. Step 202 of FIG. 2A is illustrated in blocks 240, 242, and244 of FIG. 2B, where block 240 represents the initial color image, “o”,in the sequence, block 242 represents conversion of color data to grayscale using Equation 1, and block 244 represents the image of block 240after conversion to gray scale.

[0055] Step 204 of FIG. 2A represents filtering a 256×256 portion of theinitial image, for example, a portion analogous to the 256×256 centralportion 104 of the image 102 of FIG. 1A, using Laplacian of Gaussianfiltering. Other filtering techniques are used in other embodiments.Preferred embodiments employ Laplacian of Gaussian filtering, whichcombines the Laplacian second derivative approximation with the Gaussiansmoothing filter to reduce the high frequency noise components prior todifferentiation. This filtering step may be performed by discreteconvolution in the space domain, or by frequency domain filtering. TheLaplacian of Gaussian (LoG) filter may be expressed in terms of x and ycoordinates (centered on zero) as shown in Equation (2): $\begin{matrix}{{{LoG}\left( {x,y} \right)} = {{- {\frac{1}{\pi \quad \sigma^{4}}\left\lbrack {1 - \frac{x^{2} + y^{2}}{2\sigma^{2}}} \right\rbrack}}^{- \frac{x^{2} + y^{2}}{2\sigma^{2}}}}} & (2)\end{matrix}$

[0056] where x and y are space coordinates and σ is the Gaussianstandard deviation. In one preferred embodiment, an approximation to theLoG function is used. In the embodiments described herein, approximationkernels of size 9×9, 21×21, and 31×31 are used. The Gaussian standarddeviation σ is chosen in certain preferred embodiments as shown inEquation (3):

σ=LoG filter size/8.49  (3)

[0057] where LoG filter size corresponds to the size of the discretekernel approximation to the LoG function (i.e. 9, 21, and 31 for theapproximation kernels used herein). Other embodiments employ differentkernel approximations and/or different values of Gaussian standarddeviation.

[0058] The LoG filter size may be chosen so that invalid scans arefailed and valid scans are passed with a minimum of error. Generally,use of a larger filter size is better at reducing large structured noiseand is more sensitive to larger image features and larger motion, whileuse of a smaller filter size is more sensitive to smaller features andsmaller motion. One embodiment of the invention comprises using morethan one filter size, adjusting to coordinate with the kind of motionbeing tracked and the features being imaged.

[0059] Step 204 of FIG. 2A is illustrated in FIG. 2B in blocks 244, 246,and 248, where block 244 represents data from the initial image in thesequence after conversion to gray scale intensity, block 246 representsthe application of the LoG filter, and block 248 represents the 256×256matrix of data values, G_(o)(x,y), which is the “gold standard” by whichother images are compared in validating misalignment correctiondeterminations in this embodiment. As detailed in FIG. 2C, preferredembodiments validate a misalignment correction determination bycomparing a given image to its preceding image in the sequence, not bycomparing a given image to the initial image in the sequence as shown inFIG. 2B. Although FIG. 2A, FIG. 2B, and FIG. 2C show application of theLoG filter as a discrete convolution in the space domain, resulting in astandard expressed in space coordinates, other preferred embodimentscomprise applying the LoG filter in the frequency domain. In eithercase, the LoG filter is preferably zero padded to the image size.

[0060] Steps 206 and 208 of FIG. 2A represent preprocessing an image“i”, for example, by converting RGB values to gray scale intensity asdiscussed above, and performing LoG filtering to obtain G_(i)(x,y), amatrix of values from image “i” which is compared with that of anotherimage in the sequence in order to determine a misalignment correctionbetween the two images. Steps 206 and 208 of FIG. 2A are illustrated inFIG. 2B in blocks 250, 252, 254, 256, and 258, where f_(i)(x,y) in block250 is the raw image data from image “i”, block 252 representsconversion of the f_(i)(x,y) data to gray scale intensities as shown inblock 254, and block 256 represents application of the LoG filter on thedata of block 254 to produce the data of block 258, G_(i)(x,y).

[0061] Similarly, steps 212 and 214 of FIG. 2A represent preprocessingan image “j”, for example, by converting RGB values to gray scaleintensity as discussed above, and performing LoG filtering to obtainG_(j)(x,y), a matrix of values from image “j” which is compared withimage “i” in order to determine a measure of misalignment between thetwo images. In some preferred embodiments, image “j” is subsequent toimage “i” in the sequence. In some preferred embodiments, “i” and “j”are consecutive images. Steps 212 and 214 of FIG. 2A are illustrated inFIG. 2B in blocks 264, 266, 268, 270, and 272, where “j” is “i+1”, theimage consecutive to image “i” in the sequence. In FIG. 2B, block 264 isthe raw “i+1” image data, block 266 represents conversion of the “i+1”data to gray scale intensities as shown in block 268, and block 270represents application of the LoG filter on the data of block 268 toproduce the data of block 272, G_(i+1)(x,y).

[0062] Steps 210 and 216 of FIG. 2A represent applying a Fouriertransform, for example, a Fast Fourier Transform (FFT), using G_(i)(x,y)and G_(j)(x,y), respectively, to obtain F_(i)(u,v) and F_(j)(u,v), whichare matrices of values in the frequency domain corresponding to datafrom images “i” and “j”, respectively. Steps 210 and 216 of FIG. 2A areillustrated in FIG. 2B by blocks 258, 260, 262, 272, 274, and 276, where“j” is “i+1”, the image consecutive to image “i” in the sequence. InFIG. 2B, block 258 represents the LoG filtered data, G_(i)(x,y),corresponding to image “i”, and block 260 represents taking the FastFourier Transform of G_(i)(x,y) to obtain F_(i)(u,v), shown in block262. Similarly, in FIG. 2B block 272 is the LoG filtered data,G_(i+1)(x,y), corresponding to image “i+1”, and block 274 representstaking the Fast Fourier Transform of G_(i+1)(x,y) to obtainF_(i+1)(u,v), shown in block 276.

[0063] Step 218 of FIG. 2A represents computing the cross correlationF_(i)(u,v) F*_(j)(u,v), where F_(i)(u,v) is the Fourier transform ofdata from image “i”, F*_(j)(u,v) is the complex conjugate of the Fouriertransform of data from image “j”, and u and v are frequency domainvariables. The cross-correlation of two signals of length N₁ and N₂provides N₁+N₂−1 values; therefore, to avoid aliasing problems due tounder-sampling, the two signals should be padded with zeros up toN₁+N₂−1 samples. Step 218 of FIG. 2A is represented in FIG. 2B by blocks262, 276, and 278. Block 278 of FIG. 2B represents computing the crosscorrelation, F_(i)(u,v) F*_(i+1)(u,v), using F_(i)(u,v), the Fouriertransform of data from image “i”, and F*_(i+)(u,v), the complexconjugate of the Fourier transform of data from image “i+1”. Thecross-correlation may also be expressed as c(k,l) in Equation (4):

c(k,l)=ΣΣI ₁(p,q)I ₂(p−k,q−l)  (4)

[0064] where variables (k,l) can be thought of as the shifts in each ofthe x- and y-directions which are being tested in a variety ofcombinations to determine the best measure of misalignment between twoimages I₁ and I₂, and where p and q are matrix element markers.

[0065] Step 220 of FIG. 2A represents computing the inverse Fouriertransform of the cross-correlation computed in step 218. Step 220 ofFIG. 2A is represented in FIG. 2B by block 280. The resulting inverseFourier transform maps how well the 256×256 portions of images “i” and“j” match up with each other given various combinations of x- andy-shifts. Generally, the normalized correlation coefficient closest to1.0 corresponds to the x-shift and y-shift position providing the bestmatch, and is determined from the resulting inverse Fourier transform.In a preferred embodiment, correlation coefficients are normalized bydividing matrix values by a scalar computed as the product of the squareroot of the (0,0) value of the _(auto)-correlation of each image. Inthis way, variations in overall brightness between the two images have amore limited effect on the correlation coefficient, so that the actualmovement within the image frame field between the two images is betterreflected in the misalignment determination.

[0066] Step 222 of FIG. 2A represents determining misalignment valuesd_(x), d_(y), d, sum(d_(x)), sum(d_(y)), and Sum(d_(j)), where d_(x) isthe computed displacement between the two images “i” and “j” in thex-direction, d_(y) is the computed displacement between the two imagesin the y-direction, d is the square root of the sum d_(x) ²+d_(i) ² andrepresents an overall displacement between the two images, sum(d_(x)) isthe cumulative x-displacement between the current image “j” and thefirst image in the sequence “o”, sum(d_(y)) is the cumulativey-displacement between the current image “j” and the first image in thesequence “o”, and Sum(d_(j)) is the cumulative displacement, d, betweenthe current image “j” and the first image in the sequence “o”. Step 222of FIG. 2A is represented in FIG. 2B by blocks 282, 284, and 286. Blocks284 and 286 represent finding the maximum value in the data of block 282in order to calculate d_(x), d_(y), d, sum(d_(x)), sum(d_(y)), andSum(d_(i+1)) as described above, where image “j” in FIG. 2A is “i+1” inFIG. 2B, the image consecutive to image “i”.

[0067] Steps 224, 226, and 228 of FIG. 2A represent one method ofvalidating the misalignment correction determined for image “j” in step222 of FIG. 2A. This method of validating misalignment correction isrepresented in blocks 287, 289, 291, 296, 297, and 298 of FIG. 2C.Another method of validating a misalignment correction is represented insteps 230, 232, and 234 of FIG. 2A; and this method is represented inblocks 288, 290, 292, 293, 294, and 295 of FIG. 2B. FIG. 2C is aschematic flow diagram depicting steps in a version of the methods shownin FIG. 2A of determining a correction for a misalignment between twoimages in which validation is performed using data from two consecutiveimages. Preferred embodiments comprise using consecutive ornear-consecutive images to validate a misalignment correctiondetermination, as in FIG. 2C. Other embodiments comprise using theinitial image to validate a misalignment correction determination for agiven image, as in FIG. 2B.

[0068] In FIG. 2A, step 224 represents realigning G_(j)(x,y), theLoG-filtered data from image “j”, to match up with G_(i)(x,y), theLoG-filtered data from image “i”, using the misalignment values d_(x)and d_(y) determined in step 222. In preferred embodiments, image “j” isconsecutive to image “i” in the sequence of images. Here, image “j” isimage “i+1” such that G_(i)(x,y) is aligned with G_(i+1)(x,y) as shownin block 287 of FIG. 2C. Similarly, in FIG. 2A, step 230 representsrealigning G_(j)(x,y), the LoG-filtered data from image “j”, to match upwith G_(o)(x,y), the LoG-filtered “gold standard” data from the initialimage “o”, using the displacement values sum(d_(x)) and sum(d_(y))determined in step 222. Step 230 of FIG. 2A is represented in block 288of FIG. 2B.

[0069] Step 226 of FIG. 2A represents comparing corresponding validationcells from G_(j)(x,y) and G_(i)(x,y) by computing correlationcoefficients for each cell. This is represented schematically in FIG. 2Cby blocks 289, 291, 296, 297, and 298 for the case where j=i+1. First, a128×128 pixel central portion of the realigned G_(i+1)(x,y) is selected,and the corresponding 128×128 pixel central portion of G_(i)(x,y) isselected, as shown in blocks 289 and 291 of FIG. 2C. An exemplary128×128 pixel validation region 154 is shown in FIG. 11B. Then, theembodiment comprises computing a correlation coefficient for each of 16validation cells. An exemplary validation cell from each of therealigned G_(i+1)(x,y) matrix 291 and G_(i)(x,y) matrix 289 is shown inblocks 297 and 296 of FIG. 2C. The validation cells are as depicted inthe 32×32 pixel divisions 156 of the 128×128 pixel validation region 154of FIG. 1B. Different embodiments use different numbers and/or differentsizes of validation cells. Correlation coefficients are computed foreach of the 16 cells, as shown in block 298 of FIG. 2C. Each correlationcoefficient is a normalized cross-correlation coefficient as shown inEquation (5): $\begin{matrix}{{c^{\prime}\left( {m,n} \right)} = \frac{\sum{\sum{{I_{1}\left\lbrack {p,q} \right\rbrack} \times {I_{2}\left\lbrack {p,q} \right\rbrack}}}}{\sqrt{\sum{\sum{I_{i}^{2}\left\lbrack {p,q} \right\rbrack}}}\sqrt{\sum{\sum{I_{2}^{2}\left\lbrack {p,q} \right\rbrack}}}}} & (5)\end{matrix}$

[0070] where c′(m,n) is the normalized cross-correlation coefficient forthe validation cell (m,n), m is an integer 1 to 4 corresponding to thecolumn of the validation cell whose correlation coefficient is beingcalculated, n is an integer 1 to 4 corresponding to the row of thevalidation cell whose correlation coefficient is being calculated, p andq are matrix element markers, I₁[p,q] are elements of the cell in columnm and row n of the 128×128 portion of the realigned image shown in block291 of FIG. 2C, and I₂[p,q] are elements of the cell in column m and rown of the 128×128 portion of G_(i)(x,y) shown in block 289 of FIG. 2C.Here, p=1 to 32 and q=1 to 32, and the sums shown in Equation (5) areperformed over p and q. The cross-correlation coefficient of Equation(5) is similar to an auto-correlation in the sense that a subsequentimage is realigned with a prior image based on the determinedmisalignment correction so that, ideally, the aligned images appear tobe identical. A low value of c′(m,n) indicates a mismatching between twocorresponding cells. The misalignment correction determination is theneither validated or rejected based on the values of the 16 correlationcoefficients computed in step 298 of FIG. 2C. For example, eachcorrelation coefficient may be compared against a threshold maximumvalue. This corresponds to step 228 of FIG. 2A.

[0071] Step 232 of FIG. 2A represents comparing corresponding validationcells from G_(j)(x,y) and G_(o)(x,y) by computing correlationcoefficients for each cell. This is represented schematically in FIG. 2Bby blocks 290, 292, 293, 294, and 295 for the case where j=i+1. First, a128×128 pixel central portion of the realigned G_(i+1)(x,y) is selected,and the corresponding 128×128 pixel central portion of G_(o)(x,y) isselected, as shown in blocks 292 and 290 of FIG. 2B. An exemplary128×128 pixel validation region 154 is shown in FIG. 1B. Then, theembodiment comprises computing a correlation coefficient for each of the16 validation cells. An exemplary validation cell from each of therealigned G_(i+1)(x,y) matrix 292 and G_(o)(x,y) matrix 290 is shown inblocks 294 and 293 of FIG. 2B. The validation cells are as depicted inthe 32×32 pixel divisions 156 of the 128×128 pixel validation region 154of FIG. 1B. Different embodiments use different numbers of and/ordifferent sizes of validation cells. Correlation coefficients arecomputed for each of the 16 cells, as shown in block 295 of FIG. 2B.Each correlation coefficient is a normalized “auto”-correlationcoefficient as shown in Equation (5) above, where I₁[p,q] are elementsof the cell in column m and row n of the 128×128 portion of therealigned subsequent image shown in block 292 of FIG. 2B, and I₂[p,q]are elements of the cell in column m and row n of the 128×128 portion ofG_(o)(x,y) shown in block 290 of FIG. 2B. A low value of c′(m,n)indicates a mismatching between two corresponding cells. Themisalignment determination is then either validated or rejected based onthe values of the 16 correlation coefficients computed in step 295 ofFIG. 2C. This corresponds to step 234 of FIG. 2A.

[0072] In an illustrative embodiment, determinations of misalignmentcorrection and validation of these determinations as shown in each ofFIG. 2A, FIG. 2B, and FIG. 2C are performed using a plurality of theimages in a given sequence. In preferred embodiments, determinations ofmisalignment correction and validations thereof are performed whileimages are being obtained, so that an examination in which a givensequence of images is obtained may be aborted before all the images areobtained. In some embodiments, a misalignment correction is determined,validated, and compensated for by adjusting the optical signal detectiondevice obtaining the images. In certain embodiments, an adjustment ofthe optical signal detection device is made after each of a plurality ofimages are obtained. In certain embodiments, an adjustment, if requiredby the misalignment correction determination, is made after every imagesubsequent to the first image (except the last image), and prior to thenext consecutive image. In one embodiment, a cervical tissue scancomprising a sequence of 13 images is performed using on-the-flymisalignment correction determination, validation, and cameraadjustment, such that the scan is completed in about 12 seconds. Otherembodiments comprise obtaining sequences of any number of images in moreor less time than indicated here.

[0073] Each of steps 228 and 234 of the embodiment of FIG. 2A representsapplying a validation algorithm to determine at least the following: (1)whether the misalignment correction can be made, for example, byadjusting the optical signal detection device, and (2) whether themisalignment correction determined is valid. In an exemplary embodiment,the validation algorithm determines that a misalignment correctioncannot be executed during an acetowhitening exam conducted on cervicaltissue in time to provide sufficiently aligned subsequent images, ifeither of conditions (a) or (b) is met, as follows: (a) d_(i), thedisplacement between the current image “i” and the immediately precedingimage “i−1” is greater than 0.55-mm or (b) Sum(d_(i)), the totaldisplacement between the current image and the first image in thesequence, “o”, is greater than 2.5-mm. If either of these conditions ismet, the exam in progress is aborted, and another exam must beperformed. Other embodiments may comprise the use of differentvalidation rules.

[0074] In the exemplary embodiment above, validation is performed foreach determination of misalignment correction by counting how many ofthe correlation coefficients c′_(r)(m,n) shown in Equation (5),corresponding to the 16 validation cells, is less than 0.5. If thisnumber is greater than 1, the exam in progress is aborted. Otherembodiments may comprise the use of different validation rules. Gradualchanges in image features, such as acetowhitening of tissue or changesin glare, cause discrepancies which are reflected in the correlationcoefficients of the validation cells, but which do not represent aspatial shift. Thus, in preferred embodiments, the validation isperformed as shown in FIG. 2C, where validation cells of consecutiveimages are used to calculate the correlation coefficients. In otherembodiments, the validation is performed as shown in FIG. 2B, wherevalidation cells of a current image, “i”, and an initial image of thesequence, “o”, are used to calculate the correlation coefficients ofEquation (5).

[0075]FIG. 3 depicts a subset of adjusted, filtered images 302, 306,310, 314, 318, 322 from a sequence of images of a tissue with an overlayof gridlines showing the validation cells used in validating thedeterminations of misalignment correction between the images, accordingto an illustrative embodiment of the invention. By performing validationaccording to FIG. 2C, using consecutive images to calculate thecorrelation coefficients of Equation (5), the number of validation cellswith correlation coefficient below 0.5 for the misalignment-correctedimages of FIG. 3 is 0, 1, 0, 0, and 1 for images 306, 310, 314, 318, and322, respectively. Since none of the images have more than onecoefficient below 0.5, this sequence is successful and is not aborted.This is a good result in the example of FIG. 3, since there is nosignificant tissue movement occurring between the misalignment-correctedimages. There is only a gradually changing glare, seen to move withinthe validation region 304, 308, 312, 316, 320, 324 of each image. In anembodiment in which validation is performed as in FIG. 2B, the number ofvalidation cells with correlation coefficient below 0.5 for themisalignment-corrected images of FIG. 3 is 3, 4, 5, 5, and 6 for images306, 310, 314, 318, and 322, respectively. This is not a good result inthis example, since the exam would be erroneously aborted, due only togradual changes in glare or whitening of tissue, not uncompensatedmovement of the tissue sample.

[0076] In a preferred embodiment, validation cells that are featurelessor have low signal-to-noise ratio are eliminated from consideration.These cells can produce meaningless correlation coefficients.Featureless cells in a preferred embodiment are identified andeliminated from consideration by examining the deviation of the sumsquared gradient of a given validation cell from the mean of the sumsquared gradient of all cells as shown in the following exemplary rule:

[0077] Rule: If ssg₁(m,n)<Mean[ssg(m,n)]−STD[ssg(m,n)], then setc′₁(m,n)=1.0.

[0078] where c′₁(m,n) is the correlation of the given validation cell“1”, ssg₁(m,n)=ΣΣI₁ ²[p,q], m=1 to 4, n=1 to 4, I₁[p,q] is the matrix ofvalues of the given validation cell “1”, p=1 to 32, q=1 to 32, thesummations ΣΣ are performed over pixel markers p and q, Mean[ssg(m,n)]is the mean of the sum squared gradient of all 16 validation cells, andSTD[ssg(m,n)] is the standard deviation of the sum squared gradient ofthe given validation cell “1” from the mean sum squared gradient. Bysetting c′₁(m,n)=1.0 for the given validation cell, the cell does notcount against validation of the misalignment correction determination inthe rubrics of either step 228 or step 234 of FIG. 2A, since acorrelation coefficient of 1.0 represents a perfect match.

[0079] If an image has large intensity differences between the upper andlower borders and/or the left and right borders of the image framefield, LoG filtering may result in “wraparound error.” A preferredembodiment employs an image blending technique such as “feathering” tosmooth border discontinuities, while requiring only a minimal amount ofadditional processing time.

[0080]FIG. 4A depicts a sample image 402 after application of a 9-pixelsize [9×9] Laplacian of Gaussian filter (LoG 9 filter) on an exemplaryimage from a sequence of images of tissue, according to an illustrativeembodiment of the invention. The filtered intensity values are erroneousat the top edge 404, the bottom edge 406, the right edge 410, and theleft edge 408 of the image 402. Since LoG frequency domain filteringcorresponds to cyclic convolution in the space-time domain, intensitydiscontinuities between the top and bottom edges of an image and betweenthe right and left edges of an image result in erroneous gradientapproximations. These erroneous gradient approximations can be seen inthe dark stripe on the right edge 410 and bottom edge 406 of the image402, as well as the light stripe on the top edge 404 and the left edge408 of the image 402. This often results in a misalignment correctiondetermination that is too small, since changes between the images due tospatial shift are dwarfed by the edge effects. A preferred embodimentuses a “feathering” technique to smooth border discontinuities andreduce “wraparound error.”

[0081] Feathering comprises removal of border discontinuities prior toapplication of a filter. In preferred embodiments, feathering isperformed on an image before LoG filtering, for example, between steps206 and 208 in FIG. 2A. In embodiments where LoG filtering is performedin the frequency domain (subsequent to Fourier transformation),feathering is preferably performed prior to both Fourier transformationand LoG filtering. For one-dimensional image intensity functions I₁(x)and I₂(x) that are discontinuous at x=x₀, an illustrative featheringalgorithm is as follows: $\begin{matrix}{{{I_{1}^{\prime}(x)} = {{{I_{1}(x)} \cdot {f\left( {\frac{x - x_{0}}{d} + 0.5} \right)}}\quad \text{and}}}{{{I_{2}^{\prime}(x)} = {{I_{2}(x)} \cdot \left( {1 - {f\left( {\frac{x - x_{0}}{d} + 0.5} \right)}} \right)}},{{f(x)} = \left\{ {\begin{matrix}0 & {x < 0} \\{{3x^{2}} - {2x^{3}}} & {0 \leq x \leq 1} \\0 & {x > 1}\end{matrix},} \right.}}} & (6)\end{matrix}$

[0082] where I₁′(x) and I₂′(x) are the intensity functions I₁(x) andI₂(x) after applying the feathering algorithm of Equation (6), and d isthe feathering distance chosen. The feathering distance, d, adjusts thetradeoff between removing wraparound error and suppressing imagecontent.

[0083]FIG. 4B depicts the application of both a feathering technique anda LoG filter on the same unfiltered image used in FIG. 4A. Thefeathering is performed to account for border processing effects,according to an illustrative embodiment of the invention. Here, afeathering distance, d, of 20 pixels was used. Other embodiments useother values of d. The filtered image 420 of FIG. 4B does not displayuncharacteristically large or small gradient intensity values at the topedge 424, bottom edge 426, right edge 430, or left edge 428, sincediscontinuities are smoothed prior to LoG filtering. Also, there isminimal contrast suppression of image detail at the borders. Pixelsoutside the feathering distance, d, are not affected. The use offeathering here results in more accurate determinations of misalignmentcorrection between two images in a sequence of images.

[0084] Another method of border smoothing is multiplication ofunfiltered image data by a Hamming window. In some embodiments, aHamming window function is multiplied to image data before Fouriertransformation so that the border pixels are gradually modified toremove discontinuities. However, application of the Hamming windowsuppresses image intensity as well as gradient information near theborder of an image.

[0085]FIG. 5A is identical to FIG. 4A and depicts the application of aLoG 9 filter on an exemplary image from a sequence of images of tissueaccording to an illustrative embodiment of the invention. The filteredintensity values are erroneous at the top edge 404, the bottom edge 406,the right edge 410, and the left edge 408 of the image 402.

[0086]FIG. 5B depicts the application of both a Hamming window and a LoG9 filter on the same unfiltered image used in FIG. 5A. Hamming windowingis performed to account for border processing effects, according to anillustrative embodiment of the invention. Each of the edges 524, 526,528, 530 of the image 520 of FIG. 5B no longer show the extreme filteredintensity values seen at the edges 404, 406, 408, 410 of the image 402of FIG. 5A. However, there is a greater suppression of image detail inFIG. 5B than in FIG. 4B. Thus, for this particular embodiment,application of the feathering technique is preferred over application ofHamming windowing.

[0087] A skilled artisan knows other methods of smoothing borderdiscontinuities. Another embodiment comprises removing cyclicconvolution artifacts by zero padding the image prior to frequencydomain filtering to assure image data at an edge would not affectfiltering output at the opposite edge. This technique adds computationalcomplexity and may increase processing time.

[0088]FIG. 6 depicts the determination of a misalignment correctionbetween two images using methods including the application of LoGfilters of various sizes, as well as the application of a Hamming windowtechnique and a feathering technique, according to illustrativeembodiments of the invention. Image 602 and image 604 at the top of FIG.6 are consecutive images from a sequence of images of cervix tissueobtained during a diagnostic exam, each with a pixel resolution of about0.054-mm. FIG. 6 depicts the application of four different imagefiltering algorithms: (1) Hamming window with LoG 9 filtering, (2)feathering with LoG 9 filtering, (3) feathering with LoG 21 filtering,and (4) feathering with LoG 31 filtering. Each of these algorithms areimplemented as part of a misalignment correction determination andvalidation technique as illustrated in FIG. 2A and FIG. 2C, and valuesof d_(x) and d_(y) between images 602 and 604 of FIG. 6 are determinedusing each of the four filtering algorithms. For image 602, each of thefour different image filtering algorithms (1)-(4) listed above areapplied, resulting in images 606, 610, 614, and 618, respectively, eachhaving 256×256 pixels. The four different image filtering algorithms arealso applied for image 604, resulting in images 608, 612, 616, and 620,respectively, each having 256×256 pixels. Values of (d_(x), d_(y))determined using Hamming+LoG 9 filtering are (−7, 0), expressed inpixels. Values of (d_(x), d_(y)) determined using feathering+LoG 9filtering are (−2, −10). Values of (d_(x), d_(y)) determined usingfeathering+LoG 21 filtering are (−1, −9). Values of (d_(x), d_(y))determined using feathering+LoG 31 filtering are (0, −8). All of thedisplacement values determined using feathering are close in thisembodiment, and agree well with visually-verified displacement. However,in this example, the displacement values determined using Hammingwindowing are different from those obtained using the other threefiltering methods, and result in a misalignment correction that does notagree well with visually-verified displacement. Thus, for this example,feathering works best since it does not suppress as much useful imagedata.

[0089] The effect of the filtering algorithm employed, as well as thechoice of validation rules are examined by applying combinations of thevarious filtering algorithms and validation rules to pairs of sequentialimages of tissue and determining the number of “true positives” and“false positives” identified. A true positive occurs when a badmisalignment correction determination is properly rejected by a givenvalidation rule. A false positive occurs when a good misalignmentcorrection determination is improperly rejected as a failure by a givenvalidation rule. The classification of a validation result as a “truepositive” or a “false positive” is made by visual inspection of the pairof sequential images. In preferred embodiments, whenever true failuresoccur, the scan should be aborted. Some examples of situations wheretrue failures occur in certain embodiments include image pairs betweenwhich there is one or more of the following: a large non-translationaldeformation such as warping or tilting; a large jump for which motiontracking cannot compute a correct translational displacement; rotationgreater than about 3 degrees; situations in which a target laser is lefton; video system failure such as blur, dark scan lines, or frameshifting; cases where the image is too dark and noisy, in shadow; caseswhere a vaginal speculum (or other obstruction) blocks about half theimage; other obstructions such as sudden bleeding.

[0090] In one embodiment, a set of validation rules is chosen such thattrue positives are maximized and false positives are minimized.Sensitivity and specificity can be adjusted by adjusting choice offiltering algorithms and/or choice of validation rules. Table 1 showsthe number of true positives (true failures) and false positives (falsefailures) determined by a validation rule as depicted in FIG. 2A andFIG. 2C where validation is determined using consecutive images. Table 1shows various combinations of filtering algorithms and validation rules.The four filtering algorithms used are (1) Hamming windowing with LoG 9filtering, (2) feathering with LoG 9 filtering, (3) feathering with LoG21 filtering, and (4) feathering with LoG 31 filtering. The values,c′(m,n), correspond to the normalized “auto”-correlation coefficient ofEquation (5) whose value must be met or exceeded in order for avalidation cell to “pass” in an embodiment. The “Number Threshold”column indicates the maximum number of “failed” validation cells, out ofthe 16 total cells, that are allowed for a misalignment correctiondetermination to be accepted in an embodiment. If more than this numberof validation cells fail, then the misalignment correction determinationis rejected. TABLE 1 True positives and false positives of validationdeterminations for embodiments using various combinations of filteringalgorithms and validation rules. Number c′(m,n) Threshold TP FP HammingLoG 9 −0.1 1 34 28 Feathering LoG 9 −0.1 3 19 17 Feathering LoG 21 0.3 246 10 0.35 3 52 4 Feathering LoG 31 10.5 3 48 3

[0091] For the given set of cervical image pairs on which the methodsshown in Table I were applied, feathering performs better than Hammingwindowing, since there are more true positives and fewer falsepositives. Among different LoG filter sizes, LoG 21 and LoG 31 performsbetter than LoG 9 for both tracking and validation here. The LoG 21filter is more sensitive to rotation and deformation than the LoG 31filter for these examples. Preferred embodiments for the determinationand validation of misalignment corrections between 256×256 pixelportions of images of cervical tissue with pixel resolution of about0.054-mm employ one or more of the following: (1) use of feathering forimage border processing, (2) application of LoG 21 filter, (3)elimination of validation cells with low signal-to-noise ratio, and (4)use of consecutive images for validation.

Equivalents

[0092] While the invention has been particularly shown and describedwith reference to specific preferred embodiments, it should beunderstood by those skilled in the art that various changes in form anddetail may be made therein without departing from the spirit and scopeof the invention as defined by the appended claims.

What is claimed is:
 1. A method of compensating for image misalignment,the method comprising the steps of: obtaining a sequence of images of atissue sample; and correcting for misalignment between at least two ofthe images, said misalignment being due at least in part to movement ofthe tissue sample.
 2. The method of claim 1, wherein the correcting stepis performed in real time.
 3. The method of claim 1, wherein thecorrecting step comprises adjusting an optical signal detection deviceused to obtain the sequence of images.
 4. The method of claim 3, whereinthe correcting step comprises adjusting a position of a component of theoptical signal detection device.
 5. The method of claim 4, wherein thecomponent comprises a mirror.
 6. The method of claim 1, wherein thetissue sample is an in-situ tissue sample and wherein the misalignmentis due at least in part to patient motion.
 7. The method of claim 1,further comprising the step of applying a contrast agent to the tissuesample.
 8. The method of claim 1, wherein the correcting step compriseselectronically adjusting at least one of the images.
 9. The method ofclaim 1, wherein the at least two images are consecutive images.
 10. Themethod of claim 1, wherein the correcting step comprises the step offiltering a subset of data from a first image of the sequence of images.11. The method of claim 10, wherein the correcting step comprises thestep of preprocessing the subset of data prior to the filtering.
 12. Themethod of claim 10, wherein the filtering step comprises at least one offrequency domain filtering and discrete convolution in the space domain.13. The method of claim 10, wherein the filtering step comprisesLaplacian of Gaussian filtering.
 14. The method of claim 10, wherein thefiltering step comprises using a feathering technique.
 15. The method ofclaim 10, wherein the filtering step comprises using a Hamming window.16. The method of claim 1, wherein the correcting step comprisescomputing a cross correlation using data from two of the images.
 17. Themethod of claim 16, wherein the computing of the cross correlationcomprises computing a product represented by F_(i)(u,v)F*_(j)(u,v),where F_(i)(u,v) is a Fourier transform of data derived from a subset ofdata from a first image, i, of the sequence of images, F*_(j)(u,v) is acomplex conjugate of a Fourier transform of data derived from a subsetof data from a second image, j, of the sequence of images, and u and vare frequency domain variables.
 18. The method of claim 17, wherein thecomputing of the cross correlation comprises computing an inverseFourier transform of the product.
 19. The method of claim 1, wherein thetissue sample comprises cervical tissue.
 20. The method of claim 1,wherein the tissue sample comprises at least one member of the groupconsisting of colorectal tissue, gastroesophageal tissue, urinarybladder tissue, lung tissue, and skin tissue.
 21. The method of claim 1,wherein the tissue sample comprises epithelial cells.
 22. The method ofclaim 1, wherein the obtaining step comprises obtaining the sequence ofimages of the tissue sample during application of a chemical agent tothe tissue sample.
 23. The method of claim 1, wherein the obtaining stepcomprises obtaining the sequence of images of the tissue sample afterapplication of a chemical agent to the tissue sample.
 24. The method ofclaim 23, wherein the chemical agent is selected from the groupconsisting of acetic acid, formic acid, propionic acid, and butyricacid.
 25. The method of claim 23, wherein the chemical agent is selectedfrom the group consisting of Lugol's iodine, Shiller's iodine, methyleneblue, toluidine blue, indigo carmine, indocyanine green, andfluorescein.
 26. The method of claim 1, wherein the obtaining stepcomprises obtaining the sequence of images of the tissue sample duringan acetowhitening test.
 27. The method of claim 1, wherein the movementof the tissue sample is relative to an optical signal detection deviceand comprises at least one member of the group consisting oftranslational motion, rotational motion, warping, and local deformation.28. The method of claim 1, wherein one or more images of the sequence ofimages comprise measurements of an optical signal from the tissuesample.
 29. The method of claim 28, wherein the optical signal comprisesvisible light.
 30. The method of claim 28, wherein the optical signalcomprises fluorescent light.
 31. The method of claim 28, wherein theoptical signal is emitted by the tissue sample.
 32. The method of claim28, wherein the optical signal is reflected by the tissue sample. 33.The method of claim 28, wherein the optical signal is transmittedthrough the tissue sample.
 34. A method of validating a correction foran image misalignment, the method comprising the steps of: adjusting atleast one of two or more images using a correction for an imagemisalignment between the two or more images; defining one or morevalidation cells, each of which includes a common area of the two ormore adjusted images; computing for each of the one or more validationcells a measure of displacement between the two or more adjusted imagesusing data from the two or more adjusted images corresponding to each ofthe one or more validation cells; and validating the correction for theimage misalignment by comparing at least one of the measures ofdisplacement with a threshold value.
 35. A method of validating acorrection for an image misalignment, the method comprising the stepsof: defining one or more validation cells within a bounded image plane;computing for each of the one or more validation cells a measure ofdisplacement between two or more images bound by the image plane usingdata from the two or more images corresponding to each of the one ormore validation cells; validating a correction for an image misalignmentbetween the two or more images by comparing at least one of the measuresof displacement with the correction.
 36. The method of claim 34, whereinthe images are images of an in-situ tissue sample, and wherein the imagemisalignment is due at least in part to patient motion.
 37. The methodof claim 34, wherein the images are images of an in-situ tissue samplethat has been treated with a contrast agent.
 38. The method of claim 34,wherein the one or more validation cells comprise a subset of a boundedimage plane common to the two or more images.
 39. The method of claim34, wherein the two or more images are consecutive images.
 40. Themethod of claim 38, wherein the one or more validation cells comprise acentral portion of the bounded image plane.
 41. The method of claim 38,wherein the bounded image plane has an area about four times larger thanthe total area of the one or more validation cells.
 42. The method ofclaim 34, wherein the validating step comprises eliminating fromconsideration one or more of the measures of displacement for one ormore of the one or more validation cells.
 43. The method of claim 42,wherein the eliminating of the one or more measures of displacementcomprises calculating a sum squared gradient for at least one of the oneor more validation cells.
 44. A method of compensating for an imagemisalignment, the method comprising the steps of: obtaining a set ofsequential images of a tissue sample; and correcting for a misalignmentbetween each of a plurality of pairs of the sequential images, themisalignment due at least in part to movement of the tissue sample. 45.The method of claim 44, wherein the tissue sample is an in-situ tissuesample and wherein the misalignment is due at least in part to patientmotion.
 46. The method of claim 44, further comprising the step ofapplying to the sample a contrast agent.
 47. The method of claim 44,wherein the obtaining step and the correcting step are performedalternately.
 48. The method of claim 44, wherein the obtaining step andthe correcting step are performed substantially concurrently.
 49. Themethod of claim 44, wherein the correcting step comprises determining acorrection for a misalignment between a pair of the sequential imagesless than about 2 seconds after the obtaining of the latter of the pairof the sequential images.
 50. The method of claim 44, wherein thecorrecting step comprises determining a correction for a misalignmentbetween a pair of the sequential images less than about one second afterthe obtaining of the latter of the pair of the sequential images.
 51. Amethod of validating a correction for an image misalignment, the methodcomprising the steps of: obtaining a plurality of sequential images of asample using an optical signal detection device; determining acorrection for a misalignment between at least two of the sequentialimages, the misalignment due at least in part to a movement of thesample; and validating the correction between at least a first image anda second image of the plurality of sequential images.
 52. The method ofclaim 51, wherein the sample is an in-situ tissue sample and wherein themisalignment is due at least in part to patient motion.
 53. The methodof claim 51, further comprising the step of applying a contrast agent tothe sample.
 54. The method of claim 51, wherein the determination of acorrection for a misalignment between a first and a second image and thevalidation of said correction are performed in less than about onesecond.
 55. The method of claim 51, further comprising the step of:adjusting the optical signal detection device using the correction. 56.A method of dynamically compensating for image misalignment, the methodcomprising the steps of: obtaining a sequence of images of a tissuesample; and correcting in real time for misalignment between at leasttwo of the images, the misalignment due at least in part to movement ofthe tissue sample.